DOCUMENT RESUME 



ED 435 595 



SP 038 845 



AUTHOR 

TITLE 



PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Griffin, David; LeBlanc, Patrice; McConnell , Twyla 
Initial Findings on the Benchmarks Pilot Project for the 
Nova Southeastern University Undergraduate Education 
Program . 

1999-08-09 

20p.; Paper presented at the Association of Teacher 
Educators Summer Conference (San Antonio, TX, August 7-11, 
1999) . 

Reports - Evaluative (142) -- Speeches/Meeting Papers (150) 

MFOl/PCOl Plus Postage, 

♦Benchmarking; *Educational Quality; Elementary Education; 
Higher Education; Mentors; Preservice Teacher Education; 
Program Evaluation; *Quality Control; Standards; Student 
Teachers; Undergraduate Study 
Nova Southeastern University FL 



ABSTRACT 



This paper summarizes the initial evaluation results of a 
pilot project conducted to implement program benchmarks for Nova Southeastern 
University's Undergraduate Teacher Education Program in Elementary Education. 
The program benchmarks are a response to the national movement toward 
standards to ensure teacher effectiveness. They serve as checkpoints to 
assess students' progress toward developing teaching effectiveness. The paper 
briefly describes the Elementary Education Program, its benchmarks, and the 
mentoring process used in the pilot project, A formal evaluation design of 
the pilot project used a pretest-posttest nonequivalent control group design, 
which was supplemented by an informal evaluation process. Pre-program 
questionnaires were given to elementary education majors who participated in 
the benchmarks and mentoring process and to control group students. The 
pre-program questionnaires showed no significant differences between the two 
groups. A summary of questionnaire and informal evaluation results are 
reported. Initial findings identify areas for program improvement and 
indicate that the mentoring process is valuable to students. From the 
findings, recommendations were made. (Contains 18 references.) (Author/SM) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



iTi 

ON 

iTi 

iTi 



Q 



P-l 



BMM Pilot Project 1 



Initial Findings on the 
Benchmarks Pilot Project 
for the Nova Southeastern University 
Undergraduate Education Program 



A Paper Presented at the 

Association of Teacher Educators Summer Conference 
San Antonio, Texas 
August 9, 1999 



Dr. David Griffin, Assistant Professor 
Dr. Patrice LeBlanc, Full Professor 
Mrs. Twyla McConnell, Advisor 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



©o 

h 

s 

t O 

ERIC 



Nova Southeastern University 
Farquhar Center for Undergraduate Studies 
Department of Education 
3301 College Avenue 



2 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

D This document has been reproduced as 
received from the person or organization 
originating it. 

D Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



BMM Pilot Project 2 



Ft. Lauderdale, FL 33314 

Abstract 

This article summarizes the initial evaluation results of a pilot project conducted to 
implement the program benchmarks for Nova Southeastern University’s Undergraduate Teacher 
Education Program in Elementary Education. These program benchmarks are a response to the 
national movement toward standards to insure teacher effectiveness. They serve as checkpoints 
to assess students’ progress toward developing teaching effectiveness. The article briefly 
describes the Elementary Education Program, its benchmarks, and the mentoring process used in 
the pilot project. A formal evaluation design of the pilot project used a pretest-posttest 
nonequivalent control group design, which was supplemented by an informal evaluation process. 
Preprogram questiormaires were given to Elementary Education majors who participated in the 
benchmarks and mentoring process and to control group students. The preprogram questiormaires 
showed no significant differences between the two groups. A summary of questionnaire and 
informal evaluation results are reported. Initial findings identify areas for program improvement 
and indicate that the mentoring process is valuable to students. From the findings, 
recommendations were made. 
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Recent criticisms of education in the United States created a movement toward performance- 
based outcomes, which in turn created a plethora of standards (Darling-Hammond & Sclan, 

1996; Eisner, 1995; Lewis, 1995; Roth, 1996). These standards originate from a wide range of 
sources such as state departments of education (e.g., Florida Department of Education 
Accomplished Practices), professional organizations with subject area specializations (e.g.. 

Board for Professional Teaching Standards), and groups accrediting teacher education programs 
(e.g.. National Council for Accreditation of Teacher Education). The one common thread among 
all of these standards is their purpose-increasing students’ achievement (Gandal, 1995). In order 
to achieve this purpose, the standards for teacher education tend to focus on teacher effectiveness 
as a mechanism for increasing student achievement. Research over time has identified the 
characteristics of effective teachers, which in turn have been incorporated into teacher 
preparation programs (Arends, 1998; Howey, 1996; Hunter, 1982; Woolfolk, 1998). It is these 
characteristics that are the basis of the Florida Department of Education Accomplished Practices 
(Florida Education Standards Commission, 1996). 

This history of standards and teacher education programs clearly has an impact on Nova 
Southeastern University’s Undergraduate Teacher Education Programs (NSU/UTEP). As a 
Florida State Approved Program, NSU/UTEP must address the Accomplished Practices. In 
addition, addressing the standards helps to insure program quality. Thus, the question arises: how 
do the Nova Southeastern Undergraduate Teacher Education Programs address the issues of 
standards and teacher effectiveness? This article describes how the NSU Undergraduate Teacher 
Education Program in Elementary Education addressed the standards by sharing the results of a 
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pilot project that implemented its program benchmarks. 

Context 

In order to understand the pilot project described in this article, contextual information about 
the NSU Undergraduate Teacher Education Program in Elementary Education and the pilot 
project are provided here. 

Mission of the Program 

The Undergraduate Elementary Education Program was designed to address the present and 
future needs of classroom educators. The program aims “to prepare its graduates to enter the 
teaching profession as developing professionals who are (1) sensitive to students’ academics, 
social, and emotional needs; (2) knowledgeable of effective teaching practices and technologies; 
(3) responsive to the diverse needs of all students; (4) supportive of families and community 
involvement; (5) effective communicators; (6) thoughtful and reflective educators” (Nova 
Southeastern University Undergraduate Education Department, 1998, p. 2). 

Curriculum Design 

The curriculum of the Undergraduate Elementary Education Program consists of five 
sequential blocks. The first four blocks contain four, three credit courses accompanied by clinical 
experiences. The fifth block, student internship, is a fourteen week full-time teaching experience 
in a local school supervised by a certified Clinical Educator, and a department faculty member. ' 
The Florida Department of Education Accomplished Practices and standards from various 

' A Clinical Educator is a teacher who has participated in intensive training on the Florida Performance 
Measurement System for teacher evaluation. 
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professional organizations are integrated into the courses, clinical experiences, and teaching 
experiences in the program. 

NSU Undergraduate Elementary Education Program Benchmarks 

The NSU Undergraduate Education Program Benchmarks were designed as progress 
checkpoints or indicators of a students’ progress toward completion of the program, and thus 
document their accomplishment of standards and development of teaching effectiveness. “The 
benchmark process not only allows the staff, faculty, and administration of the program to 
monitor students’ progress, but it also provides performance measures for the program, helps 
identify goals for program improvement, and ascertains students’ satisfaction with the program 
(Tucker, 1996)” (Nova Southeastern University Undergraduate Education Department, 1998, p. 
4). 

The benchmarks are organized to coincide with the sequenced blocks of courses in the 
programs. There are four benchmarks that correspond to the four blocks of courses that precede 
the internship. Within each benchmark there are specific requirements for departmental 
matriculation, standardized test completion, and portfolio development. Completion of these 
benchmarks are tracked by academic advisors and faculty mentors. Academic advisors track 
course and benchmark registration, testing, and matriculation. Faculty mentors track portfolios. 

Students may register for the first two benchmarks concurrently, but may not register for the 
third until the first benchmark is completed. Likewise, they may not register for the fourth 
benchmark until the second is completed, nor for internship until the third benchmark is 
completed. Graduation is contingent on the completion of the fourth benchmark. 
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Mentoring Component of the Benchmarks 

The mentoring component of the benchmarks is designed to provide guidance to students as 
they complete program requirements, especially the program portfolio. Furthermore, faculty 
mentors support students as they develop the competencies necessary to become effective 
beginning teachers (LeBlanc, 1998). The following paragraphs describe the mentoring process 
used in the pilot project. 

The mentors contacted students for the initial mentoring appointment, but students were 
responsible to make appointments to see the mentors at least once every eight week cycle 
following the initial appointment. At the initial mentoring appointment, a mentoring file was 
initiated. The file was used to track appointments, correspondence, and other related information. 

During the mentoring appointments, students shared their progress in the program, reviewed 
the Benchmarks Handbook (NSU Undergraduate Education Department, 1998) including the 
portfolio process, had an opportunity to pose questions, and received answers to those questions. 
In some cases, additional contacts were made via phone and email between appointments. 
Initially, students focus was on departmental matriculation. As time progressed, the focus of the 
mentoring appointments became development of the program portfolio. For example, the mentor 
and the student discussed possible frameworks for organizing the student’s portfolio. 

Pilot Project Evaluation Design 
Formal Evaluation of the Pilot Project 

The formal evaluation of the pilot project consisted of a pretest-posttest nonequivalent 
control group design. This design was selected for several reasons. First, since the main goal of 
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the pilot project was to implement the NSU Undergraduate Teacher Elementary Education 
Program Benchmarks and collect data to facilitate informed decision making about program 
development, a formative evaluation design was the natural choice (Fitz-Gibbon Sc Morris, 1987; 
Gay, 1996). In a formative evaluation the data collected are intended for use by program staff for 
the purposes of program improvement. As a result, the design requirements can be somewhat 
more “relaxed,” as in a non-randomized control group design (Fitz-Gibbon & Morris, 1987, p. 
17). Next, since it was not possible to randomly assign students to either an experimental or 
control group, the pretest-posttest nonequivalent control group design provided a viable option 
for “assessing the effect of a program” (Fitz-Gibbon Sc Morris, 1987, p. 30). Finally, the pretest- 
posttest nonequivalent control group design allows for more powerful statistical analysis of the 
data than any of the pre-experimental designs which might have been applied to the pilot project 
(Bickman Sc Rog, 1998; Gay, 1996). 

Informal Evaluation of the Pilot Project 

Given the fact that the pilot project would take two years to complete, additional informal 
evaluation needed to occur. This informal evaluation consisted of two parts; meetings and a 
survey of students. 

First, the pilot project team, which consisted of the faculty mentors and an advisor designee, 
held regular meetings to monitor the progress of the pilot project. At the meetings discussions 
were held regarding the successes and problems with the mentoring process, and solutions were 
identified to solve problems as they arose. 

The second portion of the informal evaluation was a brief survey that was developed to 



BMM Pilot Project 8 



ascertain the mentoring group’s perceptions of the mentoring process. The results from this 
survey were used as a feedback tool to help guide the faculty mentors in future mentoring efforts. 

Subject Groups 

The experimental group consisted of twelve undergraduate Elementary Education majors 
who were at the beginning of their program in the Professional and Liberal Studies (PALS) 
College^ at Nova Southeastern University. For the purposes of this report, this group will be 
referred to as the “mentoring group.” The students in the mentoring group completed pretest or 
preprogram questionnaires and were randomly assigned to one of two faculty mentors. Students 
in the mentoring group met with their faculty mentor throughout the pilot study to discuss issues 
relating to their progress in the degree program as discussed previously. 

The control group consisted of thirteen students in the PALS college who were not 
Elementary Education majors. Their majors were either Exceptional Student Education or Early 
Childhood Education. These students were selected since they were enrolled in “alternative 
versions of the program” (Fitz-Gibbon & Morris, 1987, p. 16) and thus could serve as a basis for 
comparison.^ The students in the control group were also at the beginning of their educational 
program at NSU. Those in the control group completed the preprogram questionnaires, but were 
not part of the mentoring component. At the completion of their program all students in the 

^ Nova Southeastern University’s Farquhar Center for Undergraduate Studies has two colleges. The 
Professional and Liberal Studies College is for the traditional undergraduate student, while the Career Development 
College is designed to accommodate the learner who is typically a full time working adult interested in education 
for the purposes of a career change. 

^ In the NSU Undergraduate Education Programs eight courses taken by all students. The remaining courses in 
the various programs are specific to the major, such as methods courses for specific school age populations. 
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mentoring and control groups will complete the posttest questioimaires. 

Both the mentoring and control groups were informed about the pilot and ensured 
anonymity. They were also informed that they did not have to participate, and either participating 
or not participating would have no effect on their grades. They were also informed that they 
could withdraw from the pilot study at any time. 

Methods 

Instruments 

Two questioimaires were developed as part of the benchmarks for the purpose of gathering 
data on the students. The first one, the Preprogram Questionnaire, consisted of two parts. The 
first part gathered demographic information about the students. Items were answered by making 
a check from a list of possible alternatives or filling in a blank(s). The second part of the 
Preprogram Questionnaire asked students to rate their strengths and weaknesses in a variety of 
areas related to the characteristics of effective teachers (Arends, 1998; Howey, 1996; Hunter, 
1982; Woolfolk, 1998). Thirty-eight items were presented in a five-point Likert format (strongly 
agree, agree, neutral, disagree, and strongly disagree) and the order of items was randomly 
changed to enhance validity (Fowler, 1998; Gay, 1996). 

The second questionnaire, the Preprogram Self-Assessment, consisted of forty-six items. 
These items asked students to rate their knowledge and skills on the areas addressed by the 
Florida State Department of Education’s Accomplished Practices (Florida Education Standards 
Commission, 1996), which focus on: assessment, communication, continuous improvement, 
critical thinking, diversity, ethics, human development and learning, knowledge of subject 
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matter, learning environments, planning, role of the teacher, and technology. The Preprogram 
Self-Assessment was presented in the same format as the Preprogram Questionnaire. 

Prior to administration, both questionnaires were piloted with a group of 19 students 
education students who were in a beginning course in their programs. The pilot was conducted in 
order to refine the questionnaires and receive feedback from the students regarding the 
presentation and ease of understanding and/or responding to the items on the questionnaire. 
Following the pilot, the questionnaires were revised (Fowler, 1998; Gay, 1996). 

Students in both the mentoring and control groups completed the questionnaires during class 
time in their first education course,'* with no time limits imposed. All questionnaires were 
collected by the class professor and scanned for possible errors to insure that the questionnaires 
were complete. Data were then entered for analysis using the Statistical Package for the Social 
Sciences computer program. Simple descriptive statistics were used; frequency counts and 
percentages were calculated from the questionnaire data. Since the mentoring and control group 
students have not completed their programs as yet, only the pretest results are presented and 
discussed in this article. 

As mentioned previously, a survey was developed to ascertain student’s perceptions of the 
mentoring process. Students received these surveys during mentoring appointments and 
completed them on their own directly following the appointment. The surveys were done 

'* The first education course, Exploration of the Education Profession, is completed prior to the student’s 
matriculation into the Education Department. The course provides students with the opportunity to explore 
education issues as they relate to the role of the teacher, assess whether or not the career field is the right option for 
them, and learn about the NSU Teacher Education Program. 
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anonymously to insure candid responses on the part of the students. Since the purpose of the 
survey was simple feedback for mentors, the survey was not piloted. 

Study Limitations 

As discussed previously, the nonequivalent control group design was the best option for 
collecting the formative evaluation data in the pilot project. However, this design does pose 
limitations (Gay, 1996). These limitations are discussed here. 

First, the threat to external validity could occur due to pretest- treatment interactions. 
However, although the questionnaires may alert the students to identify critically important 
information taught in the program (i.e., the characteristics of effective teachers), the effects of 
this “alert” are inconsequential since these characteristics are the basis of the program and are 
discussed at length as they connect to the content of every course in the program. In addition to 
pretest-treatment interactions, two threats to internal validity exist; that is, regression and 
selection interactions. In order to address these concerns, plans are being made for a summative 
evaluation of the NSU/UTEP program benchmarks using a stronger research design. 

Results 

Formal Evaluation of the Pilot Project 

The results of the mentoring group’s preprogram questionnaires were compared to those in 
the control group. There were no significant differences between the two groups. Thus, a second 
series of analyses were conducted to summarize the responses among the mentoring and control 
groups. This summary, which follows, is organized by categories of information asked in the 
questionnaires. 
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Demographic and Economic Data. Review of the demographic data indicates that ninety 
two percent (92%) of the mentoring group were female, eight percent (8%) were male. In the 
area of race/ethnicity, fifty eight percent (58%) of the mentoring students were white, twenty five 
percent (25%) were African-American, and seventeen percent (17%) were Hispanic. Sixty seven 
percent (67%) of students also reported being employed. 

Purpose of Education. The next series of questions asked the students why they were 
continuing their education. Students could select more than one option; those options are 
summarized here. 



■ for financial reasons 58% 

■ a requirement of their employer 67% 

■ for career development 75% 

■ to improve knowledge 75% 

■ for personal reasons 58% 



Choice of University. Students were asked why they chose Nova Southeastern University to 
continue their education. Again, their multiple responses are summarized here. 



■ 8-week terms 67% 

■ small classes 67% 

■ NSU’s reputation 58% 

■ the reputation of the faculty 1 7% 

■ NSU’s location 58% 

■ someone recommended it 33% 

■ received academic scholarship 25% 

■ received athletic scholarship 25% 



■ received financial aid package 25% 

Student Motivation. One specific question, which is related to motivation, looked at how 
many hours per week the student planned to devote to classes outside of class meeting times. 
Seventeen percent (17%) of the students indicated that they planned to devote more than ten 
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hours per week to classes. Forty two percent (42%) reported they planned to spend between one 
and five hours per week to classes. 

Use of Resources. Six questions related to community resources the students regularly use. 
They reported using newspapers (67%), journals (42%), the Internet (92%), a library (75%), 
various magazines (50%), and study groups (25%). 

Characteristics of Effective Teachers. The next series of questions asked students to rate 
their strengths in skills that have been identified as critical to becoming an effective teacher. 
These skills are listed here. 

■ speaking in front of others 

■ organizing time 

■ working as part of a group 

■ working with others from diverse populations 

■ oral communication skills 

■ written communication skills 

■ time management skills 

■ understanding of how knowledge is acquired 

■ having good interpersonal skills 

■ ability to relate to others on a one-to-one basis 

■ the ability to adjust to changes (plarmed and unplanned) in the enviromnent 
Table 1 summarizes students’ responses to these items. 



Insert Table 1 about here. 



Informal Evaluation of the Pilot Project 

As mentioned previously, due to the duration of the pilot project, an informal evaluation 
process was established to monitor the pilot project. This process included regular meetings of 
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the pilot project team and a survey completed by students. Through these processes the following 
successes and problems were identified. 

1 . Through the mentoring process, students were able to ask questions and seek support and 
guidance for their progress in the program. For example, one student came with questions 
about the framework of the portfolio, while another sought a letter of support for a 
scholarship application. 

2. Students’ feedback on the Benchmarks Handbook indicated that changes needed to be 
made to improve the clarity of the handbook, thus making it easier for students to use. 
Examples of changes in the handbook were more in depth material about matriculation 
requirements, development of an artifact cover sheet to facilitate portfolio completion, 
and reorganization of the format of the handbook. 

3. Getting students to make and/or keep appointments was identified as a major problem. It 
appeared that students did not perceive the importance of the appointments nor that of the 
benchmarks process. 



Findings 



Formal Evaluation of the Pilot Project 

The summary of the results of the preprogram questionnaires have revealed some interesting 
information about the nature of the students enrolled in the mentoring and control groups. This 
information is useful for identifying goals for program improvement. For example, questions 
related to student motivation and use of some resources indicated that students lack a realistic 
view of the work demanded at the college level and the need for using resources to support that 
work. Clearly, a goal of the program should be to help students understand the demands of a 
college education early in the program, as well as the rigorous demands of their future 
profession. 

The questionnaires also revealed data about students’ perceptions of the skills needed to 
become an effective teacher. Areas such as speaking in front of a group, organizing and 




15 



BMM Pilot Project 15 



managing one’s time, and learning to work with others from diverse backgrounds need 
development. Students also need to develop a clear philosophy of education and understand the 
various pedagogies that facilitate the process of acquiring knowledge. The data collected here 
indicate specific areas from the skills for teaching effectiveness that need to be addressed in the 
program. This information will allow faculty to address these areas of weakness in the courses 
that they teach. Finally, since the questionnaires supplied information that identified goals for 
program improvement, the formative use of the questionnaire was validated. 

Informal Evaluation of the Pilot Project 

The informal evaluation findings follow. They correspond in sequence to the results reported 
previously. 

1 . A personal relationship of mentor-mentee was developed during the mentoring process 
which provided a framework for emotional and moral support for the student (Aaronshon, 
1996). 

2. Students’ feedback in revision of the Benchmarks Handbook will make it more “user 
friendly” to students. 

3. Students may not have perceived the importance of mentoring appointments or the 
benchmarks process because were participating in a pilot project that did not mandate 
participation. 

Recommendations 

Some recommendations have been made based on the initial findings of the pilot project. 
However, the findings of the pilot project will be shared with the program administration, 
faculty, and staff through information sessions and committee meetings. This sharing will 
produce additional recommendations for improvement in the program, benchmarks, and 




16 



BMM Pilot Project 16 



mentoring process as the finding are examined from varying perspective. The initial 
recommendations follow. 

Recommendation 1 

Broaden the use of preprogram questionnaires to all students in all programs in order to 
provide formative information to program administration, faculty and staff. 

Recommendation 2 

Administer the post program questionnaires to the mentoring and control groups to provide 
information on students’ progress in meeting the standards and developing the characteristics 
of an effective teacher, identify students’ satisfaction with the program, and generate 
information for program improvement. 

Recommendation 3 

Incorporate into early course work, perhaps in Exploration of the Education Profession, the 
goal to help students understand the demands of a college education early in the program, as 
well as the rigorous demands of their future profession. 

Recommendation 4 

Continue to mentor students as it provides support to students. Also, continue informal 
evaluation of the mentoring process to ascertain other potential benefits accrued to the 
students and to obtain feedback for improvement in the benchmarks and mentoring process. 

Recommendation 5 

Make mentoring a program requirement to insure students’ follow through on appointments 
and completion of the portfolio. 

Recommendation 6 

Identify another framework for conducting mentoring which will provide stronger motivation 
for students to attend appointments. For example, as students register for benchmarks assign 
them to professors as if they were taking a course. This “mentoring course” could have 
several scheduled meetings during each term where group mentoring can occur. Group 
support may enhance students’ motivation. Furthermore, group sessions can be followed up 
with individual appointments as needed to insure the development of the personal 
relationship required for effective mentoring (Aaronshon, 1996). 
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Table 1. Percentage Response on Items Related to Skills of Effective Teaching 





Strongly 

Agree 


Agree 


Neutral 


Disagree 


Stongly 

Disagree 


Speaking in front of others 


0% 


42% 


50% 


8% 


0% 


Organizing my time 


0% 


58% 


33% 


0% 


8% 


Working as part of a group 


17% 


50% 


17% 


17% 


0% 


Working with others from 
diverse backgrounds 


0% 


33% 


58% 


8% 


0% 


Oral communication skills 


0% 


68% 


17% 


8% 


8% 


Written communication skills 


8% 


58% 


33% 


0% 


0% 


Time management skills 


0% 


50% 


42% 


0% 


8% 


I understand how knowledge is 
acquired 


0% 


33% 


50% 


17% 


0% 


I have good interpersonal skills 


33% 


50% 


17% 


0% 


0% 


I am able to relate to others on a 
one-to-one basis 


50% 


42% 


8% 


0% 


0% 


I am able to adjust to planned 
change in the environment 


8% 


83% 


8% 


0% 


0% 


I am able to adjust to unplanned 
changes in the environment 


17% 


75% 


8% 


0% 


0% 


I have a clear philosophy of 
education 


0% 


58% 


42% 


0% 


0% 



Note. Percentages are rounded. 
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